Hello, I wanted to share an issue I observed with slow mailflow on our new Exchange 2013 CU8 Hybrid environment. It is my hope that this can help someone out there, that like me, thought moving to Exchange 2013 was a huge mistake.

Brief overview of my mail servers and typical mailflow

• Hosted spam filter service --> Palo Alto firewall --> On-Prem Exchange 2010 SP3 CAS server --> On-Prem Exchange 2010 SP3 HUB transport & mailbox server <--> Hybrid Exchange 2013 cu8 (CAS&Mailbox roles)  <--> Palo Alto firewall <--> hosted exchange online mailboxes

Ever since initial setup I experienced delays with mail flow. Email would become queued at our Hybrid Exchange 2013 cu8 server when attempting to send to the exchange online hosted mailboxes, but it would eventually send. I observed several interesting items (below).

1. mail would become stuck in the on-prem Hybrid exchange 2013 server queue, and then send out of the queue in chunks , after about 20 to 30 minutes of waiting
2. when mail started to send "in chunks", all mail would be delivered out of the queue in seconds
3. after about 45 minutes or so, mail would start queuing again and repeat the process
4. On prem and external-to-our-organization mail would queue mail destined to exchange online mailboxes. Hosted mailboxes would send to other hosted mailboxes instantly, but the hosted accounts queued mail when sending back to on-prem mailboxes.

Here is what the delayed mail headers would be like (local addresses removed for my benefit)

Initially I thought it could be "the DNS bug" described in the slow-mail-flow thread over here. While I followed the steps and manually specified our DNS settings, we continued to experience the problem.

LOGS and Errors

I enabled verbose logging on the Hybrid on-prem Exchange 2013 connectors

My log file path is:
D:\Exchange2013\TransportRoles\Logs\FrontEnd
D:\Exchange2013\TransportRoles\Logs\HUB

Found errors:

\Logs\Hub\ProtocolLog\SmtpSend errors:

*,,Connector is configured to send mail only over TLS connections and remote doesn't support TLS

\Logs\Hub\Connectivity errors:

*,Session Failover; previous session id = 08D250C62FEB1479; reason = SocketError

Indeed the above errors was related to an invalid TLS certificate setup on our on-prem Exchange 2010 SP3 mailbox server. After fixing the certificate, we still experienced the slow mail queue, but had no more errors in our exchange logs.

The above troubleshooting took about a week to hammer out. During that time I asked our network engineer to take another look at the network config, and he noticed that he had set the Palo Alto firewall to allow port 25 traffic incoming traffic, but he did not allow port 25 outgoing traffic. After he changed the rule to allow outgoing port 25, our problem was gone. Somehow the firewall ended up being the issue all along, and even though plenty of Microsoft articles start with "check your firewall", I was assured that our firewall was OK and email even eventually found a way out (I have some thoughts on that - did the messages send when incoming port 25 traffic opened up? your thoughts welcome). Thanks for reading and I hope someone out there finds this information useful.

• Changed type Vinas 18 hours 38 minutes ago not a question

Thanks for sharing.